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ABSTRACT 



Traditional methods of ethnographic data collection 
and analysis can be adapted to conducting summative evaluations of 
educational programs, particularly those for American Indians. Such 
evaluations can be approached from a micro-ethnographic standpoint 
(in which the school alone is studied as a subculture) or a 
macros ethnographic standpoint, (in which the school is studied as part 
of thA culture as a whole). While each approach has advantages, the 
macro-iethnographic approach is better suited to non- Western cultures 
and American Indian "cultures in particular, because it can be used to 
discover discrepancies between the educational goals of the community 
and the\actualities of. the schools. Ethnographic evaluation methods 
are essentially the same for each approach- in an iterative rather 
than sequential process, data are collected via interviews (using 
descriptive, structural, and contrastive questions) and participant 
observatibn (to determine the places, actors, and activities of the 
program), and analyzed to determine its domains and taxonomy. 
Ethnographic summative evaluation is advantageous because it 
eliminates the evaluator's cultural values; it needs no control group 
or culturally biased achievement testing; it needs no statistical 
analysis; its reports are more useful; and it facilitates the study 
of the cultural appropriateness of the program. (SB) 
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The ethnographic method is particularly valuable for 
examining a program or process in cultural context. Activities 
are considered in relation- to sets of activities in a culture 
and according to the world view of the participants. Due to the 
holistic nature of Indian cultures, cultural subsystems 
(economics, political structure, kinship, education, religion, 
etc.) cannot be considered in isolation, but rather as highly 
interrelated. The ethnographic approach offers a framework for 
the study of Indian cultures in a holistic manner. Since the. 
cultural purpose of education is to ehcu I t urate the learner to 
become a productive member of the learner's society, the goals 
of the Indian community and the Indian I earner are inextricably 
linked. It is the purpose of this paper to present two models 
for conduxting the ethnographic summative evaluation. 

One of the fundamental distinctions made by evaluators is 
that between formative and summative evaluation (Scriven, 1967). 
The purpose of ths former is improvement of educational 
programs; that of the latter is certification, validation 6r 
OJ> overall judgement leading to decisions such as whether or not to 

so 



adopt the program. The most common way of carrying out a 
summative evaluation is to compare it with another 
statistically, after measuring the extent to which each 



$J accomplishes previously specified objectives. 

Of 

Evaluators have pointed to a number of technical difficulties 
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with the experimental approach to conducting summative 
evaluations. Random assignment frequently is impossible in 
school settings. Objectives of treatment and control programs 
frequently are not com parable; consequently tests used to 
collect achievement data often favor one program, j The 
comparison group provides no more than an a rb i t ra ry>£t a nda r d . . 
Insufficient attention is given to verifying the extent to which 
the program syl!abus was adhered to, leaving ambiguous exactly 
what must be done to achieve the same effects again. Halo 
effects, Hawthorne effects and contamination of the experimental, 
program by the control program and vice-versa frequently 
comp I i cat*e> interpretation of results: Finally, the reporting of 
results statisti cs I Ty, with numbers usually reported to two 
decimal points, gives many readers an exaggerated impression of 
the accuracy and objectivity of the method. In a word, the 
experimental approach is simplistic in its view of education, 
subtly biased by any difficulties encountered in. carrying out 
the approach under real world conditions and in effect an 
elaborate appeal to authority. 

More important perhaps than these technical di f f i cu I t i es , t h e 
administrators for whom these evaluations are done often are 
dissatisfied with the results. First, the experimental design 
requires pre-speci f ication of what is to be measured and how, 
with the problem defined before the data are co I I ect ed. Resu I ts 
vary from excellent if the probl em is defined correctly to 
inappropriate if the important questions are not addressed. Most 
important, the approach is too narrowly conceived to answer the 
wide range of questions that, must be taken into account in 

judging a program. Second, experimental approaches seldom can be 

«\ ..... ■ 
.carried out without disruption of regular schedules and very 

careful and often complicated administrative arrangements. 



The question of relative costs seldom are addressed. 
Admin istrati ve requirements in implementing and running the 
program are not taken into account. Third, the relationship of 
the new program to other elements of the curriculum, both 
horizontally and vertically, are not considered. The small 
differences in learning between. programs which evaluators 
typically discover seldom are sufficient to demonstrate that one 
program is better than the other. Finally, in view of the 
relative unimportance of the questions answered compared with 
those left unanswered, the high cost of conducting such 
evaluations seems unwarranted to administrators. 

Difficulties such as these with experimental and quasi- 
experimental met hods for conducting evaluations are among the 
reasons many have argued that broader, richer methods must be 
used instead. Among the many that have been suggested is 
ethnography, which uses a wide range of both quantitative and 
qualitative methods. In this paper, we will lim.it our attention 
to qualitative ones because we wish to emphasize the: contrast 
between ethnography and experimental methods. There are, 
however, some fundamental differences between ethnography and 
evaluation that should be understood before attempting to adapt, 
the methods of the former to the purposes of the latter.." : 

Differences Between Ethnography and Evaluation 

The most important of these differences is in determining the 
problem that is addressed by the researcher. Ethnographers are 
interested in culture, which changes slowly; evaluators address 
policy questions that typi ca I I y requi re answers without delay. 
Ethnographers, under little or no time pressure, typically allow 
the issues to be studied to emerge over relatively long periods 



of time. Eva I uators -not only work to specific short-term 
deadlines., but must address questions raised by someone else. 
Thus, ethnographers expect to learn the nuances and meanings of 
a cultural "system after years spent gaining rapport with people 
in the culture being studied. Evaluators get in, get the best 
information they can and get out quickly. Ethnographers are 
process-oriented, while evaluators concentrate on outcomes. 

One consequence of th\ese differences is that ethnographers 

often feel that their science J s being prostituted by evaluators 

f ' * 

who are using t he i r . t ec Ijjpi ques * i n new ways for new .purposes. 

But, man has a long -history of modifying. existing tools to 

achieve new purposes. Adapting ethnographic methods to 

evaluation studies requires selection of the relevant elements 

of traditional ethnographic methods and ignoring others not 

relevant to evaluation tasks. The plethora of .names such as- 

"survey ethnography" and "contract ethnography" may represent 

recognition that adaptation rather than adoption is taking 

pi ace. 

In this view, the fundamental question is whether the 
traditional methods of ethnographic data col lection and analysis 
can be adapted to conducting a summative evaluation that better 
meets the needs of a program administrator than does the more 
common experimental method. Traditionally, ethnographers ' make 
inferences from what people say, what people do and what people 
use. Thev rely on two basic tools, interview and observation, 
^he former differs from the interview used by most social 
scientists primarily in its avoidance of social science 
terminol.ogy in favor of an effort to collect and classify 
information in native terms. The latter differs from the 
observation conducted by most social scientists primarily in its 



high degree of participation tn the activities of those being 
observed. They thus are distinguished by the terms 
"ethnographic i nfervi ew u " and "participant observat i on". . 



Two Approaches to Ethnographic Summative Evaluation 

The micro- ethnographic evaluation limits the scope of the 
studyto the school. Supporters of this approach (Wilson 1977) 
•view the school as a subculture, utilizing ethnographic methods ; 
to analyze and interpret activities according to the 
parti cipant ~s reality. The resulting product of this evaluation 
is generally an ethnography describing the program activities, 
who interacted with whom, and the participants' interpretations. 
The product may address the question, "Did the program 
accomplish its goals?" by discovering the goals within the 
school, considered as a closed system. One common accomplishment 
of the school ethnography is the possible discovery of 
discrepancies between the ideal or stated behavior and the real, 
or actual behavior as reflected in activities. Through 
ethnographic interview within the school, it is possible to ask 
participants if activities in the school are connected to 
elements of the culture as a- whole. This data is limited, 
however, to the perception of a small number of participants. 

In contrast, the macrorapproach to ethnographic evaluation 
considers school interaction as interrelated with other cultural 
subsystems (Wolcott 1 975; " Everhart 1975). In other words, the 
school is not viewed as a closed system, but rather as, a part of 
the culture a s a whole. This approach takes into account the 
process of e n c u I t u r at i o n, whereby the transmission of tjiat 

6 



cultural know I edge sha red by the community occurs as part of the 
curriculum.- There are three major steps to conducting the macro 
analysis. First, a study is conducted to determine how each of 
the cultural subsystems work in the community. This involves 
observing, participating, and interviewing to identify, 
activities. The cultural meaning of acti viti e x s is identified 
according to the methodology described below, and domains of 
interaction are identified for the culture. Second, a domain 
analysis is conducted with dat.a obtaineu from activities within 
the school. And thirdly, the final analysis compares the 
domains and world view of the cu I t u re-a t- I a rge with the 
transmission of knowledge in the school. Simitar i ties and 
differences are then identified through this comparison. When 
disjunctions exist, the more common consequence is community 
opinion that the school does not reflect the social processes of 
the culture. .' 

Advantages of the micro-approach over the macro-approach - 
include lowered costs,. reduced time, and narrowly defined 
boundaries. Disadvantages of the mi cro-approach are that the 
study is more likely to be biased by the goals set by the 
educators and the scope of the study is limited. The major, and 
extremely important, advantage of the macro-approach is the 
discovery of discrepancies between the educational goals of the 
community and the actual a.ct i vities within the school. For this 
reason, the macro-approach -is more suitable to non-Western and 
American Indian cultures in particular. Where time and funds 
may. pose constraints upon the macro-approach, existing 
ethnographies can supplement. ethnographic fieldwork in the 
cu I tu r e-at- I a rge . Additional valuable resources for the study 
of American Indian cultures are linguistic and ethno-l i ngui sti c 
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studies that describe met a- I i ngu i s t i c categories, language 
structure, and world view. 

There is an important advantage of the macro-approach over 
the micro-approach that should be considered. With the micro- 
approach, if the school is studied as a subculture, then 
connections to the cu I t u r e-a t - I a r ge are minimal. In the macro- 
approach, when the. culture is examined first, followed by an 
analysis of the a ct-i-vi t i es in the school, two types of 
information are discovered. It is possible to see if there are 
no connections between activities in the culture and the 
activities fn the schools, as well as if the activities in the 
schools do not connect to the culture. Thi s . i nf o rmati on gives a 
basis for determining if items need to be added to the 
curriculum to reflect needed cultural learning, or if. there are 
items in the curriculum that need to be deleted. The pu"pose?s 
of the evaluation, advantages, and disadvantages of each 
approach should be weighed in deciding upon the more appropriate 
ethnographic evaluation. 

The foil owi ng 'methods for ethnographic evaluation apply to 
either the micro- or the. macro- approach. For the macro- 
approach, the domain analysis would be first conducted in the 
culture-at-large and then within the school; whereas, the domain 
analysis would be conducted only within the school for the 
micro-approach. The steps for analysis are basically the same, 
with more comparison utilized in the mac no- approach. 



Methods for the Ethnographic Summative Eva I uat i on 



A summative evaluation that will, meet the information needs, 
of an educational administrator will provide answers to 

« 

evaluation questions^ such as : 

* What work .is done in the program being evaluated — that is, how 
do students and teachers actually spend their time? 

* How i*s "individualized i nst ruct i on" experienced by students? 
(versus how well does individualized instruction work?) 

* How do the various types of work done in the program- relate to 
one another? 

* What are tThe major stages through which the program moves, and 
what work is done within each stage? 

* How is achievement measured or identified and what status or 
rewards does it provide? 

* How is' the program administered or controlled? 

* How do the activities in the program relate to the cultural 
activities in the community? 

As the questions asked during any evaluation will depend on 
the decisions that have to be made, this I i s t i s illustrative, 
suggestive and incomplete. The task of the evaluator is to help 
define the necessary questions, then to collect the necessary 
information and to analyze it The actual decisions are 
necessarily the task of the administrator. This is consistent 
with definitions that view the evaluator as a collector, 
analyze^ and interpreter of information, and recognize that the 
authority for decisions must remain with the administrator who 
is responsible for the program (e.g. Alkin, 1 969). V N 

Thus, an ethnographic summative evaluation will be based on 



data co I I ectea through ethnographic interviews and by 
participant observation. It will consist of interpretations of 
interrelationships among program act 'i vi ti es - and will be based on 
the language of teachers and students rather than that of social 
scientists. The interpretations probably will emphasize 
description of program progress and structure- rather than how 
much course content has been absorbed by students or what their 
attitudes care toward the subject.'" 

Methodologically, ethnographic summative evaluation involves 
ethnographic interviews and participant observation to reach 
interpretation of structure. The relationship between 
interviews and analysis is iterative rather than sequential 
(an other. characteristic that distinguishes the method from much 
social science research), and is based on three major types of 
interview qu es t i on , ., t y p es of observation, and three types of" 
analysis'. Unlike the experimental approach, it does not require 
a control group. 



Interview Questions 

The distinguishing characteristic of ethnographic interviews 
i s- the" effort" to discover questions that should be asked and the 
phrasing which they should be used. There are two main ways to 
discover the initial questions to ask. The first is to listen 
to ana record the questions people who are part of the program 
or society ask of one another about the activities. The second" 
is to inquire directly of some participants as to the types of 
questions they ask one another. 

The first of three major types of i n t e r vi ew\ qu es t i ons are 
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descriptive. These should be careful/fy phrased to avoid 
reflecting the evaluations view' of the program"."" "Could you 
describe a. typical .day *(or hour, class period, etc)"? or "Could 
you describe what' 1 happened yesterday (last hour, last period, 
etc.)" both permit informants to use the^ir own language and ' 
their own c at ego r i es ,-\_B.ot h -a.i;e LiJ$ e I y - 1 o giy^ the "eval uator • a 
general .pi cture of the p rog ram' from the respondents' 
perspectives and to introduce the vocabulary they use among 
themselves." -Both 'can lead ,to more detailed questions about 
anything descri b&d, ' and permit the evaluator very quickly to 
begin asking questions using the respondent's language. As some 
respondent s "use different language when talking to outsiders, 
it is .a wise precaution to suggest that each one answer 
questions as if t h e ' e va I u a t o r were. some specific person in the 
cu I ture, pe rhaps a new student or a substitute teacher. This 
not only will help the respondent to avoid using artificial 
language but also may provide clues to the socijal structure 
developing in the program. v 

The second of three major types of interview quest ions . are 
structural. The major purposes'^of structural questions is to 
identify the full range* of program terms and their 
interrelationships. Structural questions often must inc I ud.e an 
explanation or a context so that respondents fu I Ly understand 
what is desired; further they must be repeated to elicit all 
possible te/ms. To identify the full range of terms, questions 
such- as "Are there different kinds of (term learned from the 
descriptive questions)? are raised. To . determine relationships 
among terms, a variety of strategies are available. A. standard 

form would ask "Is a kind of (again, 

using terms learned ea r li-er ) ? " or "Would a teacher (student, 

11 



a i ci e / e t c . ) call a a kind of (still .using' 

terms learned earlier)?". A good way to get at whether things are 
Op ar e not* related is to write each term that has been learned 
on a separate 3"x5" card and to ask respondents to make as many 
^epa/ate pile's of cards as they wish based on terms that have 
Som/thing in common with one another, then to ask 'what the cards 
in ea c h pile have in common. This process can be repeated to 
'earn the full range of ways in which respondents classify the 
Same terms. 4 

'jh'e third major type of interview question seek s out contrast 
r athef than ^similarities. Again, care must be taken not to 
impose the aV&fuat or 1 s classification schemes on the 
.Respondent s .| Therefore the process .may be begun by asking 
r es po n Gents To identify any differences between terms without 
the e^ a 'tJ3tor knowing whether or not there are any. If the 
^Umbe-r °^ tennis is large, it may be wise, to present the 
Respondent with thfee terms at a time, a Wei ask which one is the 
^Ost different from the other-two and- on what basis. This is 
Effective because? tfve respondent \as to 'select the organizing 
& M n c i p I e . An extension of this is to have the informant sort 
the entire pile. of. cards- into two or more piles in terms of 
s i mi I- a r i t*i es / and then to have the informant explain the 
Rationale for the sorting. Finally, to confirm the distinctions 
k e i n g * m a d e , rr\ f f o r>m a n t s can be a s k-e d to s o r>t the cards into two 
&i| e . s based on a single specific distinction that, has^ been. 
q i scovered, 



£j rjMcl Pant O bservation 

What people s ay they do and what they actual I y do often are 
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quite different; even if this is not the case, people may not 
Itet.l an evaluator everything they do, either intentionally or 
simply because they do not think much of what they do would be. 
of any interest. Participant observation is necessary to 
complete the recorTS. 

Every soc i a r* s i tuat i on' c an be identified by three primary 
elements: a place, the actors and the activities,. In conducting 
participant observation, the evaluator will select a place; 
Determine who the actors are, watch and become involved .with 
them to some degree; and determine what their activities are. 
The evaluator will find that one "place" (e.g. the classroom) 
.consists of several distinct settings (e.g. a science table, a 
corner reserved for use of tape recorders, .student, desks and the 
teacher's supply closet). Sone' activities' may' seem independent 
of a specific place (e.g. reading in the reference-corner, at a 
student's desk, or in the library). Some activities may seem 
indepenoent of specific groups (e.g. all students might use tne. 
tape recorder at one- time or another). Some activities require 
combinations of the three. That is, a certain group of people 
in a certain place ma^y bejoget her often although their" 
activities often change, or a certain activity may, always take 
place in a certain place regardless of the group involved. 
Finally, while the actors in an educational setting usual ly Cbut 
not always) will include students and a teacher, there may be 
additional actors (e.g. aides, parents, supervisors, 
specialists) all or some of the time. ' _ ......... 
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The first task is to determine the "place" or "places" in 
hichthe program being evaluated occurs, with the aid of the 
program staff. This may seem obvious at first, but many' 
activities of many educational programs -extend far beyond the 

ERJC , ° 



classroom. Perhaps the most obvious example is the library. 
Programs that require homework of students suggest another 
area and one mo re difficult if not impossible to observe. 
Students may not only work alone but also may work in groups,, 
ana they may discuss the.program being evaluated at any time. 
Some programs may occur in several settings. For example, 
locker room, showers, playing at fields, regular classrooms, at 
"home" and "away" and the coach's office all may be places in 
which basketball programs occur. Observations will have to be 
made in as many places in which the program occurs as possible. 
Practical limitations include both access to the place and the 
total amount of time available to conduct the evaluation. 

The second task is to determine the level of participation 
the observer will have. in the situation., The educational 
e valuator must consider h ow to become involved and to what 
extent. .The possibilities vary from unobtrusive observation 
(observation without knowledge of parti cipants) through 
nonp a rt i c i p at i on (observation with knowl edge of participants but 
without any participation in activities) through complete 
Jji nvol vement (selecting a specific role and doing everything that 
role requires). Roles tnat might be selected for complete 
involvement vary from teacher ajid aide to student. Obviously, a 
major factor in making the selection will be the similarity of 

the 2 eval uator /(n age, sex, -ethnicity and other r e I ey_ant___ 

demographi c ch a p a c t e r i.s tics' to the regular program- participants. 
Other factors, such as physical condition in the case of 
evaluation of t h le ba s k e t ba I I program, may be relevant in 
particular 'situations. 

The third task is to 'determine, how observations will be 
recoxded. The requirements are to be as concrete and complete 
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as possible, and to use the terminology of the participants 
rather than those of the social scientists as much as possible. 
Ability to achieve thes* requirements depends largely on the 
length of the observati"?; session, previous experience with the 
specific type of program being evaluated and previous experience 
as a participant observer as these factors largely determine the 
likelihood of noticing and remembering observations until they 
can be recorded. Therefore, the less the experience of the 
evaluator with fieldwork, the less the experience with the 
particular type of program being evaluated, and the length of 
time after observations take place before they can be recorded, 
the less involved the evaluator should become in the program. 

The fourth task is to decide what to observe each time. 
.There are at least nine choices: specific space, actors, 
activities, ob j ects, acts, events, time, goals or feelings. The 
tendency is for initial observations to be very general, but the 
evaluator increasingly should de f i ne * sp e ci f i c purposes from 
among these nine as knowledge of the "program grows. Another 
possibility is to concentrate on interactions among the nine, 
although simultaneous observation of more than two. becomes 
unwieldy-,- Figure 1 provides a complete set of questions to 
guide observation of two way interactions among the nine. 
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SPACE 

Can you da- 
scribe in detail 
allthep/acos? 


ODJECr 

What are all the 
ways space is 
organized by 
objects? . . 


ACT 

Whal are all lha 
ways space is 
organized by 
acls? 


ACTIVirY 

Whatareallthe 
ways space is 
organized by 
activities? 


■ EVENT 

What are all lha 
ways space is 
organized by 
events? 


TIME 

What spatial 
changes occur 
overtime? 


ACTOR 

Whatareallthe 
ways space is 
used by aciors? 


. GOAL 

Whatareallthe 
ways space is 
related lo 
goals? 


FEELING 

What places are 
associated with 
feelings? 


Whore aw ob- 
iftcts located? 


Can you da* 
scribe in detail 
alllheofi/ecfs? 

■ V 

i 


Whal are all lha 

are used in 
acls? 


Whatareallthe 

wdyauujaui) 

are used in ac- 
tivities? 


Whatareallthe 

WdyS Hldl UU* 

jeclsarousedin 
events? 


How are objects 

U30U j| uiliuf- 

enl times? 


Whatareallthe. 

ways uujoLij 

are used by ac- 
tors? 


'How are objects 

U3QU III dUOnillj 

goals? 


Whatareallthe 

frays UUJ0LI3 

evoke feelings? 


Where do acls 
occur? 


i 

How doacls in- 
corporate the 
use ol objects? 


Can you de- 
scribe in detail 
Billheads? 


How are acls a 
part ol ac- 
tivities? 


How are acts a 
part ol events? 


How do acts 
vary over time? 


Whal are Ihe 
ways acts are 
performed by 
aciors? 


Whatareallthe 
ways acls are 
related lo 
goals? . 


Whatareallthe. 
ways acts are 
linked lo feel- 
ings? 


What are all the 
"places activities 
occur? 


What are all Ihe 
ways activities 
incorporate ob- 
jects? 


Whatareallthe 
wavs activities 
incorporala 
acts?. 


Can you de* 
ijcfihaln detail 

dwIIUU III uviyfl 

alllheac- 
f/VWes? 


What are all ihe 
ways act Iviiies 
are part ol 
events? 


Howdoac* . 
liviliesvaryal 
different times? 


What era all the 
ways activities 
involve aciors? 


Whal are all the 
ways activities 
involve goals? 


How do ac- 
tivities involve 
.feelings? 


What are all (he 
places events 

CiulLIi . 


Whatareallthe 
ways events in- 
corporate go* 
jecls? 


Whatareallthe 
ways events in- 
corporate acisr 


Whal are all lha 
ways events in* 
corporate ac- 
livilies? 


Can you de- 
scribe in detail 
an meevenrsf 


How do events 
occur over 

limgrlSinero 

any sequent 
ing? 


How do events 
involve the var- 

lUUa ULlUla i 


How are events 
related to 

yUdlS i 


How do events 
involve feel- 

InnQ? 


Where do time 
periods occur? 


Whatareallthe 
ways lime al- 
lects objects? 


Howdoaclslall 
into time peri* 
ods? 


How do ac- 
tivities fait into 
time periods? 


How do events 
fait into time 
periods? 


Can you de- 
scribe in detail ■ 
all the r/me pe- 
riods? 


When are all the 
limes actors are 
"onstage"? 


How are goals 
related to time 
periods? 


When are feel- 
ings evoked? 


Where do ac- 

1 A tit r\l i fiCi 

iufSpl3Cfl 

tnamsalves? 


■ 

What are all W 
ways aciors use 
objects? 

i 


Whatareallthe 
ways actors use 
acls? 


How are actors 
involved inac- 
tivities? 


How are actors 

ini»nli;aW irt 

invoiveuin 
events? 


How do actors 

WiaiKJU UVol 

lime or ^dif- 
ferent limes? 


Can you de- 

4rrihA InHAlail 

JbllUQ III Uvldll 

all the aciors? 


Which actors 

arfl link rH to 

which goals? 


What are Ihe 

fcBlinnse^Dflfi* 
enced by ac* 
tdrs? 


Whereas goals 
Miufihl and 

achieved? 


What are all lha 

way? yuaij Ml , 

volve use ol ob- 
jects? 


Whatareallthe 

itjouc n/M)lc Irt* 

WdyS yUalsilr, 

volveacls? 


What activities 

dfc QOdl SodK' 

ing or linked to 
goals? 


Whal are all ihe 
ways events are 
linked to goals? 


Which goals are 
scheduled tor 
wnicn limes? 


( How do the var- 
ious goals al- 
loc! lha various 
actors? 


Can you de- 
scribe in detail 
all the goals? 


Whatareallthe 
ways goals 
evoke feelings? 


Where do ihe 
Vanouslfidlina 

1 1 ■ Sp* *# IV**MIIH 

Itit^OCClTf? 


Wijal feelings 
lead to Ihe use- 
ol what ob- 
jects? 


What ar| all Ihe 
iajaum r*»flltnn§^~ 

affect acts? 


What are all lhe_ 

ySoyeTl n ulj nfl S 

affect ac- 
tivities? 


W^£arealljhi 
"ways feelings 
aflecl events? 


Hpw.afe.leal-— 
ingsrelaiedto 
various time . 
periods? 


Wnat are all the 
ways feelings 
involve aciors? 


WhalarethT^ 
ways feelings 
influence 
goals? 


"Can you de- 
scribe in detail 
all tliu fee/togs? 
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Data Collection 

Collection of data for ethnographic evaluations- depends on 
interviews and participant observation. These methods are 
neither independent nor sequential; Rather, they should enrich 
one another in two important ways.. First, the evaluator will 
ask better questions during interviews for having conducted 
observations, and will be more alert and understanding during x 
observations for having conducted interviews. Second, neither 
method alone provides complete understanding of what actually 
occurs in a program, who is involved in a program, or what takes 
place and what activities mean to participants in a program. It 
is not particularly important which is done first because both 
methods will be used often, but both interviews and observations 
must take place to conduct a complete evaluation. Both must 
adhere as closely as, possible to the goal of description of the 
program i part Wi plant rather. than social science terminology if 
the evaluation is to make any claim to being ethnographic in its 
met hodo I ogy . 



Analysis of Interviews 8 Observations 

v he descriptive data coJIected by interviews and obs'ef*v£ t i on 
must be analyzed to determine the. underlying meaning the 
educational settings have," for' the participants. The first task 
is to identify the activities contained in tjie data. That is, 
what constitutes separate activities in the minds of the 
participants involved must be di st.ingui shed from one another. 
The second task is to discover the relationships within the 
activities. That is, for each activity, such questions as the 
the way people transmit or learn their culture through the 
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activity must be distinguished. 

Any. activity that includes at least two or more other 
activities is termed a domain, and it is known by some cover- 
term. There always is a single semantic relationship between 
the cover term and. the included, activities/ and there is a 
boundary that distinguishes' the cover term and its included 
activities from other cover terms and thei> included activities. 
The first analytical task fo the evaluator is to identify all 
the domains that comprise the program. This requires five 
steps. The first step is to select a sample of verbatim notes 
from an interview. The second step is to look for the names of 
things mentioned. by the informant. The third step is to 

determine which, of these names might cover terms ■ that is, 

terms that describe two or more things (because the "evaluator 
often will be unsure, this process generates additional 
interview questions). The fourth step is to read through 
adoitional interviews to test the hypothesized domains. The 
fifth s t ep^s^^i_d_et_e.rmi.ae_t-h e— s-emanti c "rreT¥t~i"b7i'sTrip s among the 
terms. Spread ley (1979) has suggested nine categories into which 
most semantic relationships wil l fall, an d, ; which are. useful in 
beginning such an analysis ( See Figure 2) 
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Figure 2 Semantic Relationships for the Analysis of Domains 



I . 


Strict i n c I u s i oti 


X 


i s 


a kind of y 


2. 


Spatial 


X 


is. 


a place in Y, X is a part, of Y 


3. 


C ause-ef f ect 


X 


i s 


a result of Y, X is a cause of Y 


4. 


Rationale 


X 


i s 


.a reason for.doing y 


5. 


Location for action 


X 


i s 


a p I ace f or doi ng Y 


6. 


Function 


X 


i s 


used for Y 


7. 


Means-end 


X 


i s 


a way to do Y 


8. 


Sequence 


X 


i s 


a step (stage) in Y 


9. 


Attribution 


X 


i s 


an attribute (characteristic) of Y 



(Spradley 1979:111) 



The domain analysis is completed by selecting a single 
semantic relationship (beginning with those in Figure 2), and 
searching a_sample of i n f o rma n t * s t at emen t s for possible cover 
and included terms linked by the selected semantic relationship. 
The process is repeated until the data has been fully utilized;.. 
These relationships should assist thee valuator to develop 
structural questions for each domain (see above). These lead in 
turn to a taaxonomic analysis, which shifts the emphasi s <f rom 
merely i d^j^fxlrtg—'th-e domains to underst andiTig their internal 
structure. The first of the four steps in completing a 
taxonomic analysis is selection of a single domain for analysis. 
The second step i s* to identify the. semantic relationship that 
organizes the included terms into subsets. The third step is 
to search for possibl e subsets among the i nc I uded .'terms • The 
fourth step is to search for larger, more inclusive domains that 
might include as a subset the one with which you began. 
Finally, the fifth step is to correlate the program domains with 
domains of interaction in the program culture. Repetition of 
this five step process eventual ly : wil I exhaust a I I t h e do ma i ns 
that have been discovered and .permit construction of a tentative 
taxonomy. This can be port rayed . conveni ent I y either in an 
outline, a path diagram or a box diagram depending on personal 
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preference and-the eventual audience for the report. 

It should now be^apparent that the types of interview 
questions suggested above (descriptive, structural and contrast) 
are not arbitrary/ but are required by the types of analysis 
conaucted (domain, taxonomic), and that the relationship among 
all the parts is iterative rather than sequential. The analysis 
often provides information necessary to make the questions 

specific and appropriate, while th e__jjite.r v i e.w.s___.p.r o v id e- 1 he 

Tnformation necessary to understand the program as it is viewed, 
by participants. 

Summa ry 

' An ethnographic summative evaluation adapts the data, 
collection and analytical methods of the ethnographer to the 
purpose of aeci si on-maki ng. The two most . i mpo rt ant of these 
adaptations are advance specification of the questions to be 
addressed and specific, usually eminent , dead I i nes that preclude 
the development of rapport usually desired by ethnographers. 
Such a summative evaluation remains faithful to the traditions 
of ethnography 'in attempt in g to unde r stand" "e ducat ional programs 
from the viewpoint of the participants. Thus, one advantage of 
t a~k i n g t rre~~e t h"n"org~r apfh ~i ~c appro a c h is in eliminating the cultural 
baggage, or the values that the evaluator carries with him into 
the cultural s.etting. These values can introduce bias into the 
formulation of the evaluation questions or into the final 
recommendations. The approach presented is therefore 
particularly appropriate in explaining programs for minorities 
within any culture. 
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The ethnographic approach has the second advantage of not 
requiring a control group.- Judgements can be reacfved without 
having to set up or explain to parents why children are 
receiving different programs,, or in s i tu at i ons" where the total 
number of students does not permit such division. If the purely 
qualitative methods we have suggested are the only ones used, 
the judgement must be made without the achievement data 
characteristic of most evaluations. Thi s may be a weakness. 
First, previous work suggests that one can estimate program 
effects just as well from time on task as from tests. Second, 
testing poses spec i a I p rob I ems of cultural bias particularly for 
Indian students. Third, the statistically s i gni f i cant res u I t s 
reported from con t ro I -g roup evaluations often seem too small to 
be educationally si gni f i can't so have little impact on the 
overall judgement made about a program. Given the small number 
of programs present in most American Indian communities, the 
possibility of even obtaining a comparable control group is 
unlikely. \ , - 

A third advantage of the qualitative ethnographic approach we 

have suggested is that it does not require the often esoteric 

statistical methods. The result is an evaluation report that 

the administrator who seldom understands these methods does not 

have to take on faith. Thus, the reports themselves are more 

useful to the people for whom they are intended. 
" * . 

• A fourth advantage of the ethnographic approach is that the *• 

activities and views of the cul ture-at-l arge are linked to the 

activities within the school. This enables the discoveries of 

linkages and disjunctions in the educational process, providing' 

direction, for revision of curriculum content. The cultural. 

appropriateness of the' educational program -is assessed through 
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trse of the macro -ethnographic evaluation-. 

It appears then that ethnographic methods may be useful for 
the task of conducting summative evaluations. They are 
particularly suitable when a comprehensive understanding of the 
program and its relationship to society is the most important 
factor in judging program quality," as opposed to^ such .other — 
bases as cost or cognitive achievement. But, the decision- 
making purpose of evaluative research requires adaptation rather 
than adoption of traditional ethnographic methods. 

NOTES 

"Evaluation questions" define the purpose of the study;, 
"interview questions" are asked of participants selected from , 
the program being evaluated. 
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